{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LlamaParse Agent\n",
    "\n",
    "This demo walks through using an OpenAI Agent with [LlamaParse](https://cloud.llamaindex.ai).\n",
    "\n",
    "Status:\n",
    "| Last Executed | Version | State      |\n",
    "|---------------|---------|------------|\n",
    "| Aug-19-2025   | 0.6.61  | Maintained |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-cloud-services \"llama-index>=0.13.0<0.14.0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"llx-...\"\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "Settings.embed_model = OpenAIEmbedding(model=\"text-embedding-3-small\")\n",
    "Settings.llm = OpenAI(model=\"gpt-5-mini\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing \n",
    "\n",
    "For parsing, lets use a [recent paper](https://huggingface.co/papers/2403.09611) on Multi-Modal pretraining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://arxiv.org/pdf/2403.09611.pdf -O paper.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we can tell the parser to skip content we don't want. In this case, the references section will just add noise to a RAG system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_cloud_services import LlamaParse\n",
    "from sympy import O\n",
    "\n",
    "parser = LlamaParse(\n",
    "    parse_mode=\"parse_page_with_agent\",\n",
    "    model=\"openai-gpt-4-1-mini\",\n",
    "    high_res_ocr=True,\n",
    "    adaptive_long_table=True,\n",
    "    outlined_table_extraction=True,\n",
    "    output_tables_as_HTML=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started parsing the file under job_id cd1958b0-b260-4a63-aa74-bf829a0c125f\n",
      ".."
     ]
    }
   ],
   "source": [
    "result = await parser.aparse(\"paper.pdf\")\n",
    "documents = result.get_markdown_documents(split_by_page=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.node_parser import SentenceSplitter\n",
    "\n",
    "# Chain splitters to ensure chunk size requirements are met\n",
    "nodes = SentenceSplitter(chunk_size=2048, chunk_overlap=256).get_nodes_from_documents(\n",
    "    documents\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chat over the paper, lets find out what it is about!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import VectorStoreIndex, SummaryIndex\n",
    "\n",
    "vector_index = VectorStoreIndex(nodes=nodes)\n",
    "summary_index = SummaryIndex(nodes=nodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.agent import FunctionAgent\n",
    "from llama_index.core.tools import QueryEngineTool\n",
    "\n",
    "tools = [\n",
    "    QueryEngineTool.from_defaults(\n",
    "        vector_index.as_query_engine(\n",
    "            similarity_top_k=4,\n",
    "        ),\n",
    "        name=\"query\",\n",
    "        description=\"Send a query that requires only a subset of the top-k documents to be considered\",\n",
    "    ),\n",
    "    QueryEngineTool.from_defaults(\n",
    "        summary_index.as_query_engine(),\n",
    "        name=\"query_all_docs\",\n",
    "        description=\"Send a query that requires all documents to be considered\",\n",
    "    ),\n",
    "]\n",
    "\n",
    "agent = FunctionAgent(\n",
    "    tools=tools,\n",
    "    llm=Settings.llm,\n",
    "    system_prompt=\"You are a helpful assistant that can answer questions about the paper.\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.workflow import Context\n",
    "\n",
    "# Context to persist the agent session\n",
    "ctx = Context(agent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calling tool query_all_docs with args {'input': 'Provide the summary of the paper (concise abstract-like summary).'}\n",
      "Tool call query_all_docs({'input': 'Provide the summary of the paper (concise abstract-like summary).'}) returned This paper presents a practical recipe and empirical analysis for building high-performing multimodal large language models (MLLMs). Through systematic ablations of image encoders, vision–language connectors, and pre-training data mixtures, the work identifies key design lessons: image resolution and the number of image tokens drive the largest gains, followed by encoder capacity and pre-training data; architectural choices for the vision–language connector matter far less. Data-wise, a careful mixture of captioned images, interleaved image–text documents, and some text-only data is critical — caption data boosts zero-shot captioning, interleaved documents enable strong few-shot and text performance, and text-only data preserves language capabilities. The authors apply these lessons to scale MM1: ViT-H image encoders at high resolution feeding 144 visual tokens into decoder-only LLMs (dense and MoE variants) trained on a 45/45/10 mixture (interleaved/caption/text), for ~200k steps (~400B tokens). MM1 models (dense up to 30B, MoE up to effectively tens of billions of parameters) achieve state-of-the-art few-shot pre-training metrics and competitive supervised fine-tuning results across many established multimodal benchmarks, while exhibiting enhanced in-context learning, multi-image reasoning, and few-shot chain-of-thought capabilities. Practical training details (learning-rate scaling, unfreezing the encoder during SFT, high-resolution support via positional interpolation and sub-image decomposition) and the positive impact of synthetic caption data are reported to guide reproducing and extending these findings.\n",
      "\n",
      "================\n",
      "\n",
      "Here is a concise, abstract‑style summary of the paper:\n",
      "\n",
      "- Goal: provide a practical recipe and empirical analysis for building high‑performing multimodal LLMs (MLLMs) and identify which design choices matter most.\n",
      "- Key findings: image resolution and number of image tokens yield the largest performance gains, followed by vision‑encoder capacity and pretraining data; the specific architecture of the vision–language connector matters far less.\n",
      "- Data mix: a careful pretraining mixture is critical—captioned images boost zero‑shot captioning, interleaved image–text documents enable strong few‑shot and text performance, and some text‑only data preserves language capabilities. The authors use a 45/45/10 split (interleaved/caption/text).\n",
      "- MM1 models: applying these lessons, they scale ViT‑H encoders at high resolution producing 144 visual tokens into decoder‑only LLMs (dense up to 30B, MoE variants effectively larger), trained ~200k steps (~400B tokens).\n",
      "- Results: MM1 achieves state‑of‑the‑art few‑shot pretraining metrics and competitive supervised fine‑tuning across many multimodal benchmarks, with improved in‑context learning, multi‑image reasoning, and few‑shot chain‑of‑thought behavior.\n",
      "- Practical guidance: reportable tricks include learning‑rate scaling, unfreezing the encoder during SFT, supporting high resolution via positional interpolation and sub‑image decomposition, and the positive impact of synthetic caption data.\n",
      "\n",
      "Overall, the paper offers both empirical insights about what drives MLLM performance and a concrete, reproducible recipe (MM1) that attains strong multimodal capabilities.\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.agent import ToolCall, ToolCallResult\n",
    "\n",
    "handler = agent.run(\n",
    "    \"What is the summary of the paper that you have access to?\", ctx=ctx\n",
    ")\n",
    "async for ev in handler.stream_events():\n",
    "    if isinstance(ev, ToolCall):\n",
    "        print(f\"Calling tool {ev.tool_name} with args {ev.tool_kwargs}\")\n",
    "    elif isinstance(ev, ToolCallResult):\n",
    "        print(f\"Tool call {ev.tool_name}({ev.tool_kwargs}) returned {ev.tool_output}\")\n",
    "\n",
    "print(\"\\n================\\n\")\n",
    "\n",
    "resp = await handler\n",
    "print(resp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calling tool query_all_docs with args {'input': 'Describe in detail how the authors evaluate their work: which benchmarks and tasks they use (pretraining metrics, few-shot evaluation, supervised fine-tuning, multimodal benchmarks, in-context learning, chain-of-thought, multi-image reasoning), the metrics reported, baselines compared, and ablation studies conducted. Include mentions of training steps, model sizes, and any special evaluation setups (e.g., positional interpolation, sub-image decomposition, synthetic caption data).'}\n",
      "Tool call query_all_docs({'input': 'Describe in detail how the authors evaluate their work: which benchmarks and tasks they use (pretraining metrics, few-shot evaluation, supervised fine-tuning, multimodal benchmarks, in-context learning, chain-of-thought, multi-image reasoning), the metrics reported, baselines compared, and ablation studies conducted. Include mentions of training steps, model sizes, and any special evaluation setups (e.g., positional interpolation, sub-image decomposition, synthetic caption data).'}) returned Overview\n",
      "- Evaluation covers both pre-training (zero-/few-shot) and supervised fine-tuning (SFT) regimes, plus targeted analyses of in-context learning, multi-image reasoning, and chain-of-thought prompting. Evaluations include captioning, VQA, a set of text-only tasks (TextCore), and a wide collection of modern multimodal benchmarks. Results are reported for multiple model scales (dense 3B, 7B, 30B and MoE variants) and compared to several published baselines.\n",
      "\n",
      "Pre-training evaluation\n",
      "- Tasks and benchmarks:\n",
      "  - Image captioning: COCO (Karpathy test), NoCaps (val), TextCaps (val). Captioning use standard caption prompts and reporting.\n",
      "  - Visual question answering / text-in-image tasks: VQAv2 (testdev), TextVQA (val), VizWiz (testdev), GQA, OK-VQA (val).\n",
      "  - A text-only evaluation suite called TextCore (ARC, PIQA, LAMBADA, WinoGrande, HellaSWAG, SciQ, TriviaQA, WebQS) to measure preservation/quality of language capabilities.\n",
      "- Prompting and generation:\n",
      "  - Captioning prompt: \"{IMAGE} A photo of\" (or equivalent). VQA prompt: \"{IMAGE} Question: {QUESTION} Short answer:\".\n",
      "  - Greedy decoding until EOS or task-specific stop tokens. For captioning the newline is a stop token; for VQA additional stop tokens include \".\", \",\", \"Question\".\n",
      "  - VQA postprocessing follows the same logic used by OpenFlamingo implementations.\n",
      "- Metrics:\n",
      "  - Captioning: CIDEr (computed via nlg-eval).\n",
      "  - VQA and related QA tasks: task-appropriate accuracy metrics (reported as percentages).\n",
      "  - TextCore: aggregated scores reported to indicate text-only capabilities.\n",
      "  - Pre-training few-shot evaluation reported for 0-shot, 4-shot, and 8-shot settings (4- and 8-shot used as main few-shot points).\n",
      "- Splits and sampling:\n",
      "  - Few-shot prompts are sampled from training when available, otherwise validation, ensuring the query example is not one of the shots.\n",
      "- Scale and settings for pre-training evaluation runs:\n",
      "  - Most pre-training evaluations use smaller ablation setups: base ablation LLM = 1.2B (but some encoder ablations use a 2.9B LLM to ensure capacity).\n",
      "  - Final pre-trained models evaluated at 3B, 7B, and 30B (dense) and MoE variants (3B backbone with 64 experts; 7B backbone with 32 experts).\n",
      "- Baselines for pre-training comparisons:\n",
      "  - Flamingo (various sizes), Emu2 (14B, 37B), IDEFICS (9B, 80B), and other published pre-trained MLLMs where few-shot pre-training numbers are available.\n",
      "\n",
      "Supervised fine-tuning (SFT) evaluation\n",
      "- SFT data and setup:\n",
      "  - SFT mixture contains ≈1.45M examples: GPT-4/GPT-4V-generated instruction-response data (e.g., LLaVA-Conv/Complex, ShareGPT-4V), many academic VL datasets (VQAv2, GQA, OKVQA, A-OKVQA, COCO Captions, OCRVQA, TextCaps, DVQA, ChartQA, AI2D, DocVQA, InfoVQA, SynthDog-En), and a small internal text-only SFT set.\n",
      "  - Fine-tuning: 10k steps, batch size 256, sequence length 2048; optimizer AdaFactor with peak LR 1e-5 and cosine decay to 0. Both image encoder and LLM are unfrozen unless noted in ablations.\n",
      "- Benchmarks & aggregated evaluation:\n",
      "  - A large set of 12+ multimodal benchmarks is used for SFT evaluation, including VQAv2, TextVQA, ScienceQA-IMG, MMMU, MathVista, MME (perception/cognition splits), MMBench, SEED-Bench, POPE, LLaVA-Bench-in-the-Wild, MM-Vet, etc.\n",
      "  - Results reported per-dataset and combined into a meta-average for comparisons; the meta-average is normalized relative to a compact baseline to make metrics comparable across tasks.\n",
      "- Baselines and SFT comparisons:\n",
      "  - Compared against a range of SOTA and contemporary multimodal models after instruction tuning: LLaVA variants (1.5/NeXT), InstructBLIP, Qwen-VL, Emu2-Chat, CogVLM, Gemini family, GPT4V where available, and others. Both dense and MoE variants are compared when available.\n",
      "- High-resolution and multi-image SFT evaluation:\n",
      "  - Two techniques are used to support high-resolution inputs during SFT:\n",
      "    - Positional embedding interpolation to adapt ViT positional embeddings to larger resolutions (used to support 448×448, 560×560, 672×672, etc.).\n",
      "    - Sub-image decomposition (crop-based): for very high resolution (e.g., 1344×1344) the image is split into multiple sub-images (e.g., five 672×672 crops) that are encoded independently and concatenated as a sequence to the LLM.\n",
      "  - Default SFT evaluation results reported at an effective high resolution (1344×1344) via these strategies. Reported improvement with higher resolution (e.g., relative gains up to ~15% average when supporting 1344×1344 vs 336×336).\n",
      "- Chain-of-thought & few-shot in-context evaluation after SFT:\n",
      "  - MathVista is used to quantify few-shot chain-of-thought capability: example results show 0-shot 39.4, 4-shot 41.9, and an 8-shot mixed-resolution in-context setup achieves 44.4.\n",
      "  - Mixed-resolution in-context strategy: to fit more examples in context while managing token cost of high-resolution sub-image decomposition, some in-context examples are encoded at lower resolution and only the last N examples use full high-resolution decomposition (N=3 in reported experiments).\n",
      "\n",
      "Ablation studies and analyses\n",
      "- Overall ablation design:\n",
      "  - A compact base configuration is used for systematic ablations: ViT-L/14 image encoder (CLIP), C-Abstractor connector with 144 image tokens, pre-training mixture 45% captioned images / 45% interleaved image-text / 10% text-only, and a 1.2B decoder-only LLM for many ablations.\n",
      "  - One component changed at a time; evaluations are zero-/few-shot across the same captioning and VQA benchmarks.\n",
      "- Image encoder ablations:\n",
      "  - Compared contrastive (CLIP variants trained on DFN-5B, VeCap-300M, OpenAI CLIP) against reconstructive losses (AIM models).\n",
      "  - Resolution ablations: 224 → 336 → 378 px; clear finding that image resolution has the largest impact, followed by encoder capacity and training data composition. Increasing resolution yielded ~3% absolute boost in many metrics.\n",
      "  - Encoder size: ViT-L → ViT-H shows modest gains (typically <1% absolute).\n",
      "  - Training data for encoders: inclusion of synthetic caption data (VeCap) yields non-trivial few-shot improvements.\n",
      "  - Table-based reporting of 0-/4-/8-shot metrics for these variants.\n",
      "- Vision-language (VL) connector ablations:\n",
      "  - Connector types: average pooling (grid pooling + linear), attention pooling (learnable queries), and C-Abstractor (convolutional mapping / ResNet-based projector).\n",
      "  - Image token counts: experiments with 64 vs 144 image tokens per image.\n",
      "  - Findings: number of visual tokens and image resolution matter most; the particular connector architecture has comparatively little effect on final performance. Detailed 0/4/8-shot tables compare pooling strategies across token counts and resolutions.\n",
      "- Pre-training data mixture ablations:\n",
      "  - Systematically varied mixes of captioned image pairs vs interleaved image-text documents vs text-only data. Examples tested: 100% caption, mixtures such as 66/33, 50/50, and 0/100, and image/text-only ratios (e.g., 91/9, 86/14, 66/33).\n",
      "  - Key lessons:\n",
      "    - Interleaved documents are critical for few-shot and text-only performance; captioning data strongly lifts zero-shot captioning performance.\n",
      "    - Text-only data helps preserve/boost few-shot and text-only performance; including ~9–14% text-only yields a better balance.\n",
      "    - A final recommended pre-training mix is 45% interleaved / 45% image-caption / 10% text-only to balance zero- and few-shot capabilities.\n",
      "  - Impact of synthetic VeCap captions: even though small (~7% of caption pool), VeCap gives measurable few-shot gains (e.g., 2.4% and 4% absolute in reported settings).\n",
      "- SFT-specific ablations:\n",
      "  - Repeating data-mixture and connector ablations in the SFT context: caption-pretraining helps SFT zero-shot metrics; choice of VL connector still has limited effect though finer differences appear at high token counts; freezing vs unfreezing the image encoder matters (frozen better at lower resolution; unfrozen better for high-resolution SFT).\n",
      "- Hyperparameter and optimization ablations:\n",
      "  - Learning-rate grid searches run at small scales (models 9M, 85M, 302M, 1.2B) and 50k-step probes, then a log-linear fit extrapolated to larger model sizes. Grid-search experiments used 50k training steps for each setting.\n",
      "  - Resulting scaling rule and fitted formula for optimal peak learning rate as a function of LLM parameter count is provided and used to choose LRs for the 3B/7B/30B models (e.g., final LRs used: 6e-5 (3B), 4e-5 (7B), 2e-5 (30B)). Weight decay scaled as λ = 0.1 · η.\n",
      "- MoE (mixture-of-experts) experiments:\n",
      "  - Two MoE designs: 3B-MoE with 64 experts (∼64B total params, top-2 gating, replace every-2 layers) and 7B-MoE with 32 experts (∼47B total params, replace every-4 layers).\n",
      "  - Training used top-2 gating, load-balance loss coefficient 0.01, router z-loss 0.001, and otherwise the same hyperparameters and data mixture as the dense backbones. MoE variants show uniform improvements over dense counterparts on many SFT benchmarks.\n",
      "- Additional implementation/evaluation notes:\n",
      "  - Pre-training: models trained unfrozen for 200k steps (≈400B tokens) with batch size 512 and sequence length 4096, allowing up to 16 images per sequence and 144 tokens per image (≈1M text tokens + 1M image tokens per batch in the final setup). The pre-training mixture is fixed deterministically for reproducibility.\n",
      "  - Pre-training evaluation prompts, stop tokens, and postprocessing are standardized (greedy decoding), and detailed splits used for each benchmark are specified.\n",
      "  - SFT evaluation meta-average: benchmarks are normalized to a compact baseline configuration prior to averaging so disparate metrics can be compared.\n",
      "  - For high-resolution SFT, the positional interpolation approach (to support larger patches) and the sub-image decomposition scheme (to represent very large images as multiple crops) are both used and evaluated; sub-image decomposition increases the number of image tokens dramatically, which motivates mixed-resolution in-context examples for few-shot prompting.\n",
      "\n",
      "Reporting and comparisons\n",
      "- Tabular reporting:\n",
      "  - Pre-training few-shot results are reported in detailed tables per model scale (3B, 7B, 30B) for 0/4/8/16-shot where applicable, across captioning and VQA datasets.\n",
      "  - SFT comparisons show per-benchmark numbers and a combined meta-average; both dense and MoE model variants are included.\n",
      "- Baselines and contemporaries cited for direct comparison include Flamingo, IDEFICS, Emu2, LLaVA-NeXT, CogVLM, Gemini family, GPT4V, and many instruction-tuned MLLMs. Where appropriate, notes on differences in prompting setups (e.g., some baselines include text-only demonstrations in “0” prompts) are documented.\n",
      "- Qualitative analysis:\n",
      "  - A variety of qualitative examples shown for counting, OCR, multi-image reasoning, style following, instruction following, and chain-of-thought reasoning; these accompany quantitative results to illustrate capabilities such as multi-image reasoning and few-shot chain-of-thought.\n",
      "\n",
      "Key reported evaluation figures (examples)\n",
      "- Pre-training duration: 200k steps (~400B tokens).\n",
      "- Pre-training batch & context: batch 512, sequence length 4096, up to 16 images per sequence, 144 tokens per image.\n",
      "- SFT: 10k steps; batch 256; seq length 2048; AdaFactor with peak LR 1e-5.\n",
      "- MoE variants: 3B backbone + 64 experts (∼64B total); 7B backbone + 32 experts (∼47B total); top-2 gating; load-balance and router regularizers used.\n",
      "- Example few-shot chain-of-thought: MathVista 0-shot 39.4 → 4-shot 41.9 → 8-shot with mixed-resolution 44.4.\n",
      "\n",
      "In summary\n",
      "- Evaluation is multi-faceted: systematic pre-training zero-/few-shot tests on captioning and VQA, text-only TextCore checks, extensive SFT across a broad benchmark suite, ablations covering image encoder, VL connector, data mixtures, training hyperparameters, and input-resolution strategies, plus experiments with MoE scaling. Metrics include CIDEr for captioning, accuracy for VQA and other benchmarks, TextCore aggregated scores, and a normalized meta-average for SFT. The authors report results across multiple model sizes and variants and compare to a broad set of recent multimodal models.\n",
      "\n",
      "================\n",
      "\n",
      "Short answer: the authors evaluate across (1) pre-training zero-/few-shot benchmarks (captioning, VQA, and a text-only suite), (2) supervised instruction fine‑tuning (SFT) on a large multimodal mixture with extensive downstream benchmarks, and (3) targeted analyses (in‑context/few‑shot learning, chain‑of‑thought, multi‑image reasoning). They report standard task metrics (CIDEr for captioning, accuracy for VQA/QA, aggregated TextCore scores, and a normalized SFT meta‑average), compare to many recent MLLMs, and run systematic ablations (encoder, connector, data mixtures, hyperparameters, resolution/tokenization, MoE). Key training/eval settings and special setups are also evaluated (positional interpolation, sub‑image decomposition, synthetic caption data). Details:\n",
      "\n",
      "1) Pre‑training evaluation\n",
      "- Tasks and datasets:\n",
      "  - Image captioning: COCO (Karpathy test), NoCaps (val), TextCaps (val).\n",
      "  - VQA/text‑in‑image: VQAv2 (testdev), TextVQA, VizWiz, GQA, OK‑VQA, etc.\n",
      "  - TextCore: a text‑only suite (ARC, PIQA, LAMBADA, WinoGrande, HellaSWAG, SciQ, TriviaQA, WebQS) to check language preservation.\n",
      "- Prompting & decoding:\n",
      "  - Zero/4/8 (and sometimes 16) shot prompts; few‑shot examples sampled from train/val ensuring no leakage.\n",
      "  - Greedy decoding with task‑specific stop tokens; VQA postprocessing matches Flamingo style.\n",
      "- Metrics:\n",
      "  - CIDEr for captioning, accuracy (%) for VQA/QA tasks, aggregated TextCore scores for language capability.\n",
      "- Model scales for evaluation:\n",
      "  - Ablations often use a small base LLM (1.2B, sometimes 2.9B). Final pre‑trained models evaluated at 3B, 7B, 30B (dense) and MoE variants.\n",
      "- Baselines:\n",
      "  - Compared against Flamingo, Emu2, IDEFICS, and other published pre‑trained MLLMs when few‑shot pretraining numbers are available.\n",
      "\n",
      "2) Supervised fine‑tuning (SFT) evaluation\n",
      "- SFT data:\n",
      "  - ≈1.45M instruction examples: GPT‑4/GPT‑4V synthetic instruction data (LLaVA‑Conv/Complex, ShareGPT‑4V), many academic VL datasets (VQAv2, GQA, OKVQA, COCO Captions, TextCaps, OCRVQA, ChartQA, DocVQA, etc.), and a small internal text SFT set.\n",
      "- Fine‑tuning procedure:\n",
      "  - 10k steps, batch 256, seq length 2048, AdaFactor optimizer, peak LR 1e‑5 with cosine decay. Image encoder and LLM unfrozen unless ablated.\n",
      "- Downstream benchmarks and reporting:\n",
      "  - 12+ multimodal benchmarks for SFT evaluation (VQAv2, TextVQA, ScienceQA‑IMG, MMMU, MathVista, MME, MMBench, SEED‑Bench, POPE, LLaVA‑BiW, MM‑Vet, etc.). Results reported per dataset and combined into a normalized meta‑average for fair aggregation across heterogeneous metrics.\n",
      "- Baselines:\n",
      "  - Compared to instruction‑tuned contemporaries: LLaVA/NeXT, InstructBLIP, Qwen‑VL, Emu2‑Chat, CogVLM, Gemini family, GPT4V where available.\n",
      "\n",
      "3) Targeted analyses (in‑context learning, CoT, multi‑image)\n",
      "- In‑context/few‑shot: standard 0/4/8‑shot probes across captioning and VQA.\n",
      "- Chain‑of‑thought: MathVista used to quantify few‑shot CoT; reported example: 0‑shot 39.4 → 4‑shot 41.9 → 8‑shot mixed‑resolution 44.4.\n",
      "- Multi‑image reasoning: evaluated qualitatively and quantitatively on multi‑image benchmarks and examples.\n",
      "\n",
      "4) Ablation studies (systematic and extensive)\n",
      "- Image encoder ablations:\n",
      "  - Contrastive (CLIP variants) vs reconstructive (AIM); encoder size (ViT‑L → ViT‑H); encoder training data (including synthetic caption data VeCap).\n",
      "  - Resolution ablations (e.g., 224 → 336 → 378 px): resolution and number of visual tokens give the largest gains.\n",
      "- Vision–language connector ablations:\n",
      "  - Connector types (avg‑pooling, attention pooling, C‑Abstractor) and visual token counts (e.g., 64 vs 144). Finding: connector architecture matters far less than token count/resolution.\n",
      "- Pre‑training data mixture ablations:\n",
      "  - Varied mixes of caption pairs / interleaved image–text documents / text‑only. Key finding: 45% interleaved / 45% caption / 10% text gives the best balance (interleaved documents help few‑shot/text performance; captions boost zero‑shot captioning; text-only preserves language capabilities).\n",
      "  - Small synthetic caption pool (VeCap) provides measurable few‑shot gains.\n",
      "- SFT ablations:\n",
      "  - Freezing vs unfreezing image encoder in SFT (unfreeze better for high‑resolution), data‑mix effects in SFT, connector behavior at high token counts.\n",
      "- Hyperparameter & optimizer ablations:\n",
      "  - LR grid searches at small scales (9M → 1.2B) with 50k‑step probes and a fitted scaling rule; final LRs chosen (e.g., ~6e‑5 for 3B, 4e‑5 for 7B, 2e‑5 for 30B for pretraining). Weight decay scaled proportionally.\n",
      "- MoE experiments:\n",
      "  - Two MoE setups: 3B backbone + 64 experts (~64B params) and 7B + 32 experts (~47B params), top‑2 gating, load‑balance/reg losses; MoE variants yield uniform improvements on many SFT benchmarks.\n",
      "\n",
      "5) Special evaluation/training setups and numbers\n",
      "- Pretraining infrastructure & settings:\n",
      "  - Pretraining: ≈200k steps (~400B tokens), batch 512, seq length 4096, allow up to 16 images per sequence, 144 tokens per image in final setup. Pretraining mixture fixed deterministically.\n",
      "- High‑resolution support:\n",
      "  - Positional embedding interpolation to adapt ViT positional embeddings to larger resolutions.\n",
      "  - Sub‑image decomposition (split very large images into multiple crops, encode independently, and concatenate visual tokens) to support extremely high effective resolution (e.g., 1344×1344 as five 672×672 crops).\n",
      "  - Mixed‑resolution in‑context strategy to keep context capacity reasonable while enabling high‑resolution targets in the last few shots.\n",
      "- Decoding/postprocessing:\n",
      "  - Greedy decoding; task‑specific stops; standardized postprocessing to align with prior work.\n",
      "- Reporting conventions:\n",
      "  - 0/4/8‑shot pretraining tables, SFT per‑dataset numbers and a normalized meta‑average, and qualitative examples (counting, OCR, style following, multi‑image reasoning, CoT).\n",
      "\n",
      "6) Qualitative analysis\n",
      "- Numerous qualitative examples illustrating multi‑image reasoning, counting, OCR, instruction following, and chain‑of‑thought behaviors accompany the quantitative results.\n",
      "\n",
      "In short: the evaluation is broad (pretraining few‑shot, SFT, targeted capability probes), quantitatively rigorous (CIDEr/accuracy/meta‑averages), compares to many contemporary MLLMs, and is supported by wide ablations (encoder, connector, data, optimization, resolution, MoE) and practical high‑resolution evaluation techniques (positional interpolation, sub‑image decomposition, mixed‑resolution in‑context).\n"
     ]
    }
   ],
   "source": [
    "handler = agent.run(\"How do the authors evaluate their work?\", ctx=ctx)\n",
    "async for ev in handler.stream_events():\n",
    "    if isinstance(ev, ToolCall):\n",
    "        print(f\"Calling tool {ev.tool_name} with args {ev.tool_kwargs}\")\n",
    "    elif isinstance(ev, ToolCallResult):\n",
    "        print(f\"Tool call {ev.tool_name}({ev.tool_kwargs}) returned {ev.tool_output}\")\n",
    "\n",
    "\n",
    "print(\"\\n================\\n\")\n",
    "\n",
    "resp = await handler\n",
    "print(resp)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
